1. Technical Field
The present invention relates to the internet and, more particularly, to a task-crawling system and method for a distributed crawler system.
2. Description of Related Art
With the development of technologies regarding the internet, the number of websites is growing in geometric series. The way search engines crawling websites has been evolved from a single-node manner, a multi-node manner to currently a distributed manner, so as to crawling websites fast, thereby enriching the engines' indexing data and providing better user experience. Since there are many crawling nodes in a distributed crawler system, a central controlling node is required to distribute tasks to corresponding crawling nodes. In this process, there are cases where a crawl task is not properly performed due to, for example, web problems or abnormal operation of the corresponding crawling node. The present invention thus provides a highly reliable solution for distribution and execution of crawl tasks.